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ABSTRACT 

Today,  a  large  body  of  research  exists  regarding  the  correct¬ 
ness  of  routing  protocols.  However,  many  reported  global 
disruptions  of  Internet  connectivity,  e.g.,  inter-AS  persistent 
loops,  cannot  be  explained  by  looking  at  a  single  routing 
protocol  at  a  time.  In  fact,  these  anomalies  have  long  been 
suspected  in  the  operator  community  to  be  caused  by  the 
interactions  between  routing  protocols.  The  interactions  be¬ 
tween  protocol  instances  are  governed  by  two  procedures  at 
the  border  routers:  route  selection  (RS)  ranks  routes  from 
different  protocol  instances;  and  route  redistribution  (RR) 
exchanges  routes  between  protocol  instances.  Prior  studies 
hypothesized  that  RR  may  be  responsible  for  a  portion  of 
the  observed  anomalies.  In  this  paper,  we  provide  analytical 
and  experimental  results  to  link  RS,  RR,  and  their  interplay 
to  anomalies  discovered  in  operational  networks.  We  show 
that  RS  by  itself  can  cause  route  oscillations  and  loops,  and 
that  in  all  Cisco,  Quagga,  and  XORP  implementations,  non- 
deterministic  behaviors  may  occur  because  of  their  incor¬ 
rect  modeling  of  the  dependencies  between  RS  and  RR.  We 
identify  the  root  cause  for  each  of  the  instabilities  and  de¬ 
rive  a  configuration  guideline  as  well  as  a  functional  model 
to  eliminate  them. 

1.  INTRODUCTION 

One  of  the  primary  goals  of  a  network  is  to  ensure  the 
proper  delivery  of  packets  to  the  intended  destinations.  Rout¬ 
ing  protocols  play  an  essential  role  toward  that  objective. 
They  disseminate  routing  information  and  allow  routers  to 
compute  their  forwarding  tables.  Because  of  their  impor¬ 
tance,  a  large  body  of  research  has  been  devoted  to  the  cor¬ 
rectness  of  routing  protocols. 

However,  existing  analytical  frameworks  as  well  as  empir- 
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ical  studies  for  understanding  routing  dynamics  concentrate 
on  one  routing  protocol  at  a  time,  most  notably  BGP  [22], 
[33],  [19],  [6],  [18],  [16].  The  reality  is  that  a  large  num¬ 
ber  of  the  reported  disruptions  of  Internet  connectivity,  such 
as  the  ones  listed  below,  cannot  be  easily  explained  by  the 
misbehavior  of  a  single  routing  protocol. 

Persistent  forwarding  loops:  Several  studies  [31],  [34] 
have  reported  the  existence  of  persistent  forwarding  loops 
within  an  AS  or  across  multiple  networks.  The  discoveries 
of  inter-AS  routing  loops  were  surprising  given  the  fact  that 
BGP  has  been  specifically  designed  to  avert  the  formation  of 
such  loops  via  checking  the  AS  PATH  attribute.  Those  stud¬ 
ies  [31],  [34]  conjectured  that  the  interactions  between  static 
routes  and  BGP  may  have  originated  the  inter-AS  loops.  Our 
own  discussions  with  operators  reveal  that  the  operational 
community  also  views  the  interactions  between  BGP  and 
IGPs  as  a  possible  root  cause  of  this  problem:  the  injec¬ 
tion  of  routes  from  BGP  into  IGP,  and  then  re-injection  of 
the  same  routes  from  IGP  back  into  BGP  at  another  location 
will  reset  the  AS  PATH  attribute  and  render  the  BGP’s  guard 
against  inter-AS  loops  ineffective. 

IP  prefix  hijacks:  Prefix  hijacks,  such  as  the  notorious 
AS  7007  incident  [30],  periodically  happen  in  the  Internet. 
This  type  of  anomaly  can  have  a  large  impact,  disrupting  the 
connectivity  of  thousands  of  networks.  Misconfigurations 
are  frequently  cited  as  the  source  of  the  problem.  McPherson 
[29]  took  a  closer  look  at  the  root  cause  of  a  recent  IP  prefix 
hijack  and  hypothesized  that  the  interactions  between  BGP 
and  static  routes  may  have  been  at  the  origin  of  the  routing 
anomaly.  Our  discussions  with  operators  suggest  that  a  se¬ 
quence  of  route  injections  between  BGP  and  IGPs,  which 
resets  the  AS  PATH,  may  also  cause  a  network  to  advertise  a 
prefix  it  does  not  own.  Traffic  for  the  affected  prefix  is  subse¬ 
quently  black-holed  when  the  offending  network  has  insuf¬ 
ficient  resources  to  handle  the  increased  amount  of  traffic. 

Nondeterministic  forwarding  paths:  Messages  posted 
on  bulletin  boards  used  by  network  operators  indicate  that 
a  network  configuration  consisting  of  multiple  routing  do¬ 
mains  may  result  in  unexpected  forwarding  paths.  Chen  and 
Yuan  [7]  described  a  case  involving  the  interactions  between 
iBGP  and  static  routes.  Our  experiments  show  that  the  scope 
of  the  problem  is  significantly  larger:  e.g.,  interactions  be- 
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tween  BGP  and  OSPF,  or  between  OSPF  and  RIP,  have  the 
same  problem.  None  of  the  existing  analytical  frameworks 
can  explain  the  observed  outcomes. 

The  operational  community  has  long  suspected  that  the 
culprit  of  these  anomalies  may  lie  in  the  interactions  be¬ 
tween  routing  protocols.  However,  despite  the  severity  of 
the  problems,  there  is  a  surprisingly  small  number  of  studies 
on  such  interactions  from  the  research  community.  Two  of 
them  [20],  [32]  introduced  several  frameworks  to  analyze  the 
impact  of  the  underlying  IGP  on  BGP,  and  formulated  con¬ 
ditions  that  may  lead  to  forwarding  loops,  route  oscillations, 
and  delayed  convergence.  In  our  previous  work  [27],  we  de¬ 
veloped  a  formal  model  to  reason  about  the  consequences  of 
injecting  routes  across  routing  domains.  In  particular,  that 
model  can  be  used  to  explain  the  permanent  inter-AS  for¬ 
warding  loops  and  IP  prefix  hijacks  mentioned  above. 

Yet,  these  studies  cannot  make  sense  of  all  the  anomalies 
described  above.  For  example,  none  of  them  can  provide 
a  good  explanation  of  the  reported  nondeterministic  routing 
behaviors.  In  addition,  the  results  in  [20]  and  [32]  are  spe¬ 
cific  to  the  interactions  between  BGP  and  an  IGP.  However, 
recent  empirical  studies  [28],  [25]  show  that  operational  net¬ 
works  frequently  deploy  multiple  instances  of  IGP  and  join 
them  through  route  redistributions  rather  than  with  BGP  to 
achieve  important  design  objectives  such  as  efficient  routing. 
The  interactions  between  these  IGP  instances  require  further 
research. 

The  interactions  between  different  instances  of  routing  pro¬ 
tocols  configured  on  one  network,  which  we  will  simply  re¬ 
fer  to  as  routing  instances ,  are  currently  governed  by  two 
procedures  executed  at  border  routers:  route  selection  and 
route  redistribution.  The  route  selection  procedure  ranks 
routes  received  from  different  routing  protocol  instances  and 
selects  a  “best”  route  among  them  for  forwarding  purposes, 
and  the  route  redistribution  procedure  facilitates  the  exchange 
of  routing  information  between  routing  instances.  They  are 
critically  important  for  two  reasons.  First,  they  allow  opera¬ 
tors  to  fulfill  a  necessary  function,  that  of  integrating  multi¬ 
ple  routing  domains.  Second,  operators  make  extensive  use 
of  route  selection  and  route  redistribution  as  primitives  to 
achieve  important  design  objectives  that  cannot  be  accom¬ 
plished  by  routing  protocols  (including  BGP)  alone  [25], 

Prior  studies  [11],  [12],  [27]  have  provided  some  evidence 
based  on  simple  scenarios  that  route  redistribution  is  a  sepa¬ 
rate  risk  factor  from  routing  protocols  for  routing  anomalies. 
In  this  paper,  we  present  analytical  and  experimental  results 
to  link  route  selection  and  route  redistribution  to  reported 
routing  instabilities  in  operational  networks,  including  the 
aforementioned  forwarding  loops  and  nondeterministic  for¬ 
warding  paths.  We  consider  this  a  primary  contribution  of 
the  paper.  Additional  contributions  from  this  paper  are  as 
follows: 

1.  We  show  that  the  problem  is  more  fundamental  than  pre¬ 
viously  reported.  We  show  that  route  selection  by  itself  - 

i.e.,  merely  the  presence  of  multiple  routing  instances  in 


one  network,  without  any  exchange  of  routes  between  the 
routing  instances  -  can  also  result  in  some  of  the  reported 
routing  anomalies. 

2.  We  show  that  the  problem  is  broader  than  previously  re¬ 
ported.  We  present  experimental  results  showing  that  all 
tested  Cisco,  Quagga,  and  XORP  products  have  incor¬ 
rectly  implemented  the  dependencies  between  route  se¬ 
lection  and  route  redistribution  which  can  cause  nonde¬ 
terministic  routing  outcomes. 

3.  We  conduct  a  root  cause  analysis  of  the  disclosed  insta¬ 
bilities.  We  identify  necessary  conditions  for  each  cate¬ 
gory  of  anomalies  (loops,  oscillations,  nondeterministic 
routing  behaviors)  that  can  derive  from  route  selection 
and  its  interplay  with  route  redistribution.  Our  analy¬ 
sis  indicates  that  the  nondeterministic  routing  outcomes 
likely  result  from  a  lack  of  a  detailed  functional  model  of 
the  dependencies  between  route  selection  and  route  re¬ 
distribution. 

4.  Finally,  we  propose  a  configuration  guideline  to  miti¬ 
gate  some  of  the  instabilities.  We  formally  prove  that  the 
guideline  will  prevent  the  targeted  instabilities.  We  also 
present  a  functional  model  to  precisely  define  the  depen¬ 
dencies  between  route  selection  and  route  redistribution. 

The  rest  of  the  paper  is  organized  as  follows.  Section  2 
provides  more  details  on  how  the  route  selection  and  route 
redistribution  procedures  work  and  describes  two  key  prop¬ 
erties  of  their  functionality.  Section  3  analyzes  the  routing 
anomalies  due  to  route  selection.  Section  4  addresses  the 
additional  instabilities  caused  by  the  interplay  between  route 
selection  and  route  redistribution.  Section  5  presents  related 
work  and  finally.  Section  6  concludes  and  discusses  future 
work. 

2.  ABSTRACTING  THE  INTERACTIONS 

A  router  can  run  multiple  routing  protocols  (e.g.,  BGP, 
EIGRP,  IS-IS,  OSPF,  RIP)  at  the  same  time.  Certain  ven¬ 
dors  even  allow  a  router  to  create  multiple  instances  of  the 
same  routing  protocol  (e.g.,  OSPF  1,  OSPF  2).  A  software 
process  is  associated  with  each  of  the  created  routing  pro¬ 
tocol  instances  and  it  is  commonly  referred  to  as  a  routing 
process.  Each  routing  process  is  generally  assigned  a  Rout¬ 
ing  Information  Base  (RIB)  [13].  This  database  is  used  to 
store  the  routing  information  related  to  the  routing  process 
(e.g.,  routes  received  from  peers). 

2.1  Route  Selection 

A  router  may  run  multiple  routing  processes  and  receive 
more  than  one  route  (e.g.,  an  OSPF  route  and  a  RIP  route) 
to  the  same  destination  prefix.  Some  examples  and  motiva¬ 
tions  for  such  scenarios  are  described  in  Section  3.  When 
receiving  multiple  routes  to  the  same  destination  prefix  from 
different  routing  processes,  the  router  uses  an  inter-protocol 
route  selection  procedure  to  choose  one  of  the  routes  to  put 
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Figure  1:  An  enterprise  with  two  office  branches,  each 
deploying  its  own  routing  protocol.  By  default,  the  RIP 
routers  have  no  visibility  of  the  destinations  in  the  OSPF 
domain,  and  vice-versa. 

in  its  Forwarding  Information  Base  (FIB).  This  route  selec¬ 
tion  procedure  is  the  focus  of  our  study.  To  add  flexibility 
to  the  procedure,  router  vendors  have  introduced  the  con¬ 
cept  of  administrative  distance  (AD)  [14]  to  aid  ranking  of 
routes  from  different  routing  protocols.  Each  routing  pro¬ 
cess  has  a  default  AD  value  (e.g.,  1 10  for  OSPF  and  120  for 
RIP  on  Cisco  routers),  which  can  be  overridden  per  router 
and  per  prefix  with  special  router  configuration  commands. 
All  routes  by  default  inherit  the  AD  value  of  their  respec¬ 
tive  routing  process  and  the  functionality  of  the  route  se¬ 
lection  procedure  can  be  precisely  defined  by  the  following 
property: 

Route  Selection  Property  (PI):  When  multiple  routing 
processes  offer  routes  to  the  same  destination  prefix,  the  route 
with  the  lowest  AD  value  is  selected  for  the  FIB. 

The  routing  process  with  the  lowest  AD  value  is  referred 
to  as  the  selected  routing  process ,  and  the  route  that  is  put  in 
the  FIB  (to  forward  traffic)  the  active  route. 

More  specifically,  each  routing  process  first  determines  its 
best  path  using  a  protocol  specific  algorithm.  For  example, 
RIP  prefers  routes  with  the  lowest  metric  value  while  BGP 
compares  multiple  criteria  including  the  LOCAL JPREF,  the 
AS  PATH  length  and  other  parameters.  Then,  each  routing 
process  presents  its  most  preferred  route  to  the  route  selec¬ 
tion  procedure,  which  compares  all  the  received  routes  and 
chooses  the  one  with  the  lowest  AD  value. 

To  illustrate  the  route  selection  procedure,  consider  the 
network  depicted  in  Ligure  1 .  We  focus  on  router  A  and  we 
assume  that  it  is  configured  with  a  static  route  to  a  destina¬ 
tion  prefix  P.  Router  A  runs  a  routing  process  of  RIP  and  a 
routing  process  of  OSPL,  and  we  assume  that  both  are  con¬ 
figured  with  a  lower  AD  value  than  that  of  the  static  route. 
When  router  A  receives  a  route  to  destination  P  through  a 
RIP  neighbor,  A  shall  prefer  the  RIP  route  to  the  static  route 
and  use  it  to  forward  the  traffic. 

2.2  Route  Redistribution 

Routing  processes  of  different  routing  protocols  by  de¬ 
fault  are  totally  independent  and  do  not  exchange  routing 
information  even  when  they  are  running  on  the  same  router 
(e.g.,  OSPL  process  and  RIP  process  on  router  A  of  Ligure  1.) 


In  fact,  routing  processes  of  the  same  routing  protocol  on  the 
same  router  by  default  do  not  exchange  routing  information 
either  (e.g.,  OSPL  1  and  OSPL  2  on  a  same  router).  How¬ 
ever,  routing  processes  are  required  to  exchange  routing  in¬ 
formation  with  their  peer  processes,  which  are  configured  for 
the  same  routing  protocol  instance  but  on  different  routers 
(e.g.,  in  Ligure  1,  RIP  process  on  X  and  RIP  process  on  A). 
More  precisely,  two  routing  processes  are  said  to  belong  to 
the  same  routing  instance  when  they  run  on  different  routers 
and  form  an  adjacency,  i.e.,  run  the  same  routing  protocol 
and  exchange  routing  information. 

When  a  network  is  composed  of  multiple  routing  instances, 
routes  may  need  to  be  exchanged  across  routing  instances. 
By  default,  routing  information  originated  in  a  routing  in¬ 
stance  (i.e.,  by  a  member  routing  process)  remains  within  the 
boundaries  of  that  routing  instance  (i.e.,  shared  only  among 
routing  processes  of  that  routing  instance).  Lor  example,  in 
the  network  depicted  in  Ligure  1 ,  the  RIP  routers  do  not  have 
visibility  of  the  destinations  in  the  OSPL  instance  and  vice- 
versa.  To  allow  communications  across  routing  instances, 
vendors  have  introduced  a  router  function  called  route  re¬ 
distribution,  which  must  be  explicitly  enabled.  The  function 
can  be  enabled  between  any  pair  of  routing  processes  (e.g., 
one  RIP  and  the  other  OSPL)  running  on  the  same  router  to 
move  routes  from  one  (called  source)  into  the  other  (called 
target).  Although  not  formally  specified  by  vendors,  a  key 
property  for  route  redistribution  is: 

Route  Redistribution  Property  (P2):  A  route  is  advertised 
and  redistributed  only  if  it  is  active  [27]. 

A  routing  protocol  should  advertise  a  route  only  if  the 
route  is  active.  Violations  of  this  property  can  result  in  rout¬ 
ing  anomalies  as  illustrated  in  Section  3.2.  We  note  that 
by  definition,  link-state  routing  protocols  relay  all  received 
routing  information  independently  of  whether  a  route  is  ac¬ 
tive.  However,  all  vector  protocols  ought  to  satisfy  property 
P2. 

Lollowing  the  same  reasoning,  a  route  should  only  be  re¬ 
distributed  from  a  source  routing  process  into  a  target  rout¬ 
ing  process  when  the  route  is  active.  Lor  example,  consider  a 
router  running  three  routing  processes  u,  v  and  w.  Suppose 
that  redistributions  from  u  to  v  and  from  v  to  w  are  con¬ 
figured.  In  addition,  assume  that  the  active  route  has  come 
from  u.  In  such  a  case,  the  route  is  redistributed  into  v  but 
not  into  w. 

Linally,  a  recent  study  shows  that  operators  use  route  se¬ 
lection  and  route  redistribution  to  not  only  interconnect  rout¬ 
ing  instances,  but  also  meet  important  operational  needs  that 
cannot  be  provided  by  routing  protocols  alone  [25]. 

3.  INSTABILITIES  OF  ROUTE  SELECTION 

This  section  presents  routing  anomalies  that  can  derive 
from  route  selection  by  itself,  i.e.,  without  any  route  redis¬ 
tribution  configured  between  the  routing  instances.  Section 
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Figure  2:  Illustration  of  route  oscillations.  The  same  destination  prefix  P  is  originated  in  both  instances.  The  dashed 
arrows  represent  signaling  messages.  Routers  shaded  in  dark  represent  routers  with  a  route  to  the  destination.  The 
state  at  t3  is  identical  to  that  at  t6. 


3.1  illustrates  how  route  oscillations  and  forwarding  loops 
may  occur.  Section  3.2  analyzes  the  root  cause  for  each  of 
these  anomalies.  Finally,  Section  3.3  presents  a  configura¬ 
tion  guideline  to  eliminate  these  anomalies. 

Topologies  and  routing  designs  similar  to  those  described 
in  this  section  have  been  observed  in  operational  networks 
[28],  [11],  [25],  In  addition,  we  have  validated  all  the  de¬ 
scribed  scenarios,  with  the  exception  of  those  requiring  spe¬ 
cific  race  conditions,  as  it  is  difficult  to  create  them.  The 
validation  environment  consisted  of  Cisco  2600  routers  (IOS 
Version  12.2). 

3.1  Illustration  of  Routing  Anomalies 

We  use  the  following  notation  throughout  the  paper.  Rout¬ 
ing  instances  are  numbered  1,  2, ...,  routers  labeled  A,  B , ..., 
and  routing  processes  denoted  by  <router>.<routing  instance>. 
For  example,  B.  1  designates  the  routing  process  from  rout¬ 
ing  instance  1  at  router  B. 

3.1.1  Route  Oscillations 

We  assume  the  network  depicted  in  Figure  2(a).  It  is  sim¬ 
ilar  to  the  network  depicted  in  Figure  1  with  the  following 
difference:  we  assume  that  each  office  branch  deploys  an  in¬ 
stance  of  a  distance  vector  protocol,  specifically  EIGRP.  Em¬ 
pirical  studies  have  shown  that  operational  networks  com¬ 
monly  deploy  multiple  instances  of  a  same  routing  proto¬ 
col  [11],  [25].  We  further  assume  that  routers  A  and  B 
are  not  configured  to  perform  route  redistribution,  and  have 
the  default  AD  values  for  the  two  EIGRP  routing  processes. 

The  discussion  is  in  respect  to  a  given  destination  prefix  P 
originated  by  both  instances.  Recent  empirical  studies  have 
also  shown  that  in  operational  settings,  one  destination  pre¬ 
fix  can  be  originated  by  multiple  instances  [25],  P  may  be 
the  default  route,  originated  by  each  instance.  Alternatively, 
each  office  branch  may  receive  routes  to  the  same  destina¬ 
tion  through  their  respective  ISP. 

The  following  sequence  of  events  illustrates  the  possible 


formation  of  a  route  oscillation. 

1 1  Routes  are  propagated  in  both  instances  of  EIGRP. 

t2  Upon  receiving  a  route,  routers  D  and  Y  learn  a  route  to 
P  and  then  further  advertise  the  route  to  their  neighbors 
(i.e.,  A  and  B  respectively). 

t3  Router  A  receives  a  route  from  routing  process  A.  2  point¬ 
ing  to  D  as  the  next-hop.  Similarly,  router  B  receives  a 
route  from  routing  process  B.  1  pointing  to  Y  as  the  next- 
hop. 

G  Router  A  further  advertises  the  route  into  routing  instance 
2  through  its  neighbor  router  C.  At  the  same  time,  router 
B  further  advertises  the  route  into  routing  instance  1  through 
its  neighbor  router  A. 

t§  Router  A  receives  2  routes  (from  A .  1  and  .4.2).  Because 
they  have  the  same  AD  values,  some  implementations  se¬ 
lect  the  latest  received  information  [15],  i.e.,  the  route 
from  Al.  Similarly,  router  B  receives  2  routes  (from 
B.  1  and  11.2).  Because  they  have  the  same  AD  values,  B 
may  select  the  route  from  B.2. 

te  Since  router  A  selected  the  route  from  Al,  A. 2  stops  ad¬ 
vertising  a  route  to  P.  In  the  same  way,  router  B.  1  stops 
advertising  a  route  to  P.  Consequently,  routers  A  and  B 
lose  their  routes  from  Al  and  11.2  respectively.  A  reverts 
to  using  its  previous  route  from  A. 2  and  in  the  same  man¬ 
ner,  B  uses  the  route  from  II.  1  [13].  The  resulting  state 
is  identical  to  that  at  1 4.  In  other  words,  we  have  a  route 
oscillation. 

The  duration  of  the  oscillation  varies  depending  upon  how 
long  routing  events  at  routers  A  and  B  are  synchronized. 
Thus,  such  routing  anomalies  may  have  been  diagnosed  as 
transient  forwarding  loops.  We  note  that  the  described  sce¬ 
nario  consists  of  only  two  routing  instances.  Studies  [28], 
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Figure  3:  Illustration  of  a  permanent  forwarding  loop 
(A-B-C-A).  Routers  A  and  B  each  receive  two  routes 
with  identical  AD  values.  The  route  selection  is  nonde- 
terministic  and  if  A  selects  the  route  from  OSPF  1  and 
B,  the  route  from  OSPF  2,  the  loop  results. 

[25]  have  disclosed  that  operational  networks  frequently  de¬ 
ploy  dozens  or  even  hundreds  of  routing  instances.  In  such 
large  settings,  the  interactions  would  become  significantly 
more  complex  and  the  chances  for  routing  anomalies  would 
increase  considerably.  In  addition,  the  problem  can  be  ex¬ 
acerbated  by  the  proprietary  nature  of  the  AD  concept:  each 
router  vendor  has  its  own  set  of  default  values  for  routing 
protocols,  and  consequently,  route  selection  configurations 
can  result  in  not  only  oscillations  of  arbitrary  time  length 
but  also  permanent  route  oscillations  [26]. 

While  vendors  have  introduced  proprietary  solutions  such 
as  RIB  quarantining  [8]  to  mitigate  the  impact  of  route  os¬ 
cillations,  it  is  necessary  to  fix  the  problems  at  their  roots  in 
order  to  restore  connectivity.  Prior  studies  of  route  oscilla¬ 
tions  have  focused  on  BGP  [6],  [20],  [19],  [18].  We  have  just 
shown  that  the  simple  co-existence  of  IGP  instances,  even 
without  BGP,  can  be  another  plausible  cause. 

3.1.2  Forwarding  Loops 

In  addition  to  causing  route  oscillations,  the  route  selec¬ 
tions  across  multiple  protocol  instances  can  also  result  in 
permanent  forwarding  loops.  This  is  the  case  when  some  of 
the  protocol  instances  perform  link-state  routing.  We  assume 
the  network  depicted  in  Figure  3.  The  topology  is  identical 
to  those  previously  considered.  However,  each  of  the  of¬ 
fice  branches  is  now  running  OSPF.  Empirical  studies  [25] 
provide  evidence  that  operational  networks  deploy  multiple 
instances  of  OSPF  for  administrative  reasons:  each  office 
branch  may  be  administered  by  a  separate  team.  Such  de¬ 
sign  may  also  result  from  a  company  merger  or  may  be  in¬ 
tentional  in  order  to  control  the  dissemination  of  the  routes 
[11],  We  further  assume  that  no  route  redistribution  is  con¬ 
figured  and  all  the  routers  use  the  default  administrative  dis¬ 
tances  for  the  routing  processes. 

As  depicted  in  Figure  3,  this  configuration  can  result  in 
a  permanent  forwarding  loop  {A-B-C-A):  when  router  A 
receives  two  routes  (from  A.l  and  A. 2),  because  they  have 
identical  AD  values,  A  may  select  the  route  from  A.l  [11]. 
Similary,  B  may  select  the  route  from  B.  2  resulting  in  a  for¬ 
warding  loop  A-B-C-A. 

The  depicted  configuration  nondeterministically  results  in 
a  forwarding  loop  because  of  the  random  selection  at  the  bor- 


Figure  4:  Illustration  of  the  dispute  wheel  responsible  for 
the  oscillations  observed  in  Figure  2. 

der  routers,  which  increases  the  difficulties  of  the  debugging 
task.  This  scenario  shows  that  route  selection  by  itself  can 
be  at  the  origin  of  observed  permanent  forwarding  loops.  In 
addition,  B.  1  (respectively  A. 2)  may  be  configured  with  a 
strictly  larger  AD  value  than  that  of  B. 2  (respectively  A.l). 
Such  an  AD  assignment  can  happen  in  a  multi-vendor  en¬ 
vironment,  or  result  from  a  configuration  error,  and  it  will 
consistently  result  in  a  forwarding  loop. 

3.2  Analysis  of  Root  Cause 

Route  oscillations:  The  route  oscillations  described  pre¬ 
viously  (e.g.,  in  Figure  2)  occur  because  routers  repeatedly 
advertise  and  withdraw  a  route.  This  happens  in  response 
to  a  prefered  route  being  offered  and  then  retracted.  Since 
routers  in  a  link-state  protocol  advertise  all  of  their  informa¬ 
tion  -  independently  of  their  selected  paths  to  a  destination 
-  the  interactions  of  route  selections  between  only  link-state 
routing  processes  do  not  cause  route  oscillations.  Route  os¬ 
cillations  can  occur  only  if  the  route  selections  involve  a 
minimum  of  two  routing  protocol  instances  like  BGP,  RIP, 
and  EIGRP,  which  only  advertise  routes  that  are  active. 

The  root  cause  of  the  oscillations  is  in  fact  similar  to  those 
in  the  BGP  context.  Griffin  et  al.  [18]  demonstrated  that  the 
presence  of  dispute  wheels  can  be  responsible  for  route  oscil¬ 
lations.  Before  presenting  the  formal  definition  of  a  dispute 
wheel,  we  first  introduce  some  notations.  [18]  suggested 
the  following  definitions:  Considering  a  simple,  undirected 
graph,  G  =  (V,E),  where  V  is  the  set  of  nodes  and  E  the 
set  of  edges,  we  focus  on  a  specific  node,  called  the  ori¬ 
gin.  Every  other  node  attempts  to  establish  a  path  to  the 
origin.  A  path  in  G  is  either  empty  or  a  sequence  of  nodes 
(ukUk-i-.-Uo)  such  that  for  alii  G  [0,  k—  1], {uj+i,  u*}  G  E. 
Then,  for  every  u  G  V,  Vu  represents  the  set  of  permitted 
paths  from  u  to  the  origin.  Finally,  for  each  u  G  V,  there  is 
a  ranking  function  A'“,  defined  over  Vu:  if  Pi,  P2  €  Vu  and 
A“(Pi)  <  A“(P2),  then  u  prefers  the  path  P2  over  Pi. 

A  dispute  wheel  n  =  (U,  Q,  R)  of  size  k  is  defined  as  a 
sequence  of  nodes  U  =  Uq,  Ui,  ...,  Uk-i,  and  sequences  of 
non-empty  paths  Q  =  Q0,  Q 1, ...,  Qk- 1,  R  =  Ro,Ri,  •••,  Pfc-i 
such  that  for  every  i  G  [0,  k  —  1],  (1)  Ri  is  a  path  from  Ui 
to  ul+ 1;  (2)  Qi  G  P“-;  (3)  RiQi+1  G  PUi;  (4)  A u'(Qt)  < 
\Ui(RiQi+i).  (All  subscripts  are  modulo  k .) 

Figure  4  highlights  the  dispute  wheel  responsible  for  the 
route  oscillations  described  in  Section  3.1.1  and  Figure  2. 
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Forwarding  loops:  The  permanent  forwarding  loops  oc¬ 
cur  because  of  deflections.  Deflections  are  formally  defined 
[20]  as  follows:  Considering  a  destination  router  Uq,  and  as¬ 
suming  that  un  has  selected  the  forwarding  path  P(un)  = 
unun—  i . . .  no,  P(un)[ui,  Uj ]  denotes  the  subpath  of  P(un ) 
starting  at  u,  and  ending  at  Uj  with  i  >  j.  A  deflection  hap¬ 
pens  on  P(un )  at  Ui  if  P(ui )  ^  P{un)  [u,;,  «o]  but  for  all 
j  >  i ,  P(uj)  =  P(un)[uj,  u0\. 

It  was  shown  that  the  occurrence  of  deflections  between 
iBGP  and  its  underlying  IGP  can  result  in  forwarding  loops 
[20],  We  have  just  shown  that  deflections  can  result  from 
the  interaction  between  any  two  routing  instances,  causing 
permanent  loops.  In  Figure  3,  deflections  at  routers  A  and  B 
are  responsible  for  the  observed  loop.  In  fact,  the  forward¬ 
ing  loops  occur  because  of  the  AD  assignment  at  the  differ¬ 
ent  routers:  the  configuration  includes  a  dispute  wheel.  Yet, 
as  link-state  protocols  advertise  all  routes  they  receive,  re¬ 
gardless  if  they  are  active,  dispute  wheels  do  not  cause  route 
oscillations  but  deflections. 

3.3  Guideline  for  Safe  Route  Selection 

Because  the  focus  of  this  paper  is  on  the  interactions  be¬ 
tween  routing  protocols,  we  assume  that  packet  forwarding 
within  each  routing  instance  is  free  of  instabilities',  more 
formally,  the  routing  protocol  converges  and  the  forward¬ 
ing  paths  for  each  destination  form  a  directed  acyclic  graph 
where  all  routers  of  the  routing  instance  are  connected,  and 
all  the  leaf  node(s),  i.e.,  node(s)  with  no  outgoing  edges, 
either  are  directly  connected  to  the  destination  network  or 
run  multiple  routing  processes  (i.e.,  serve  as  a  border  router 
joining  multiple  routing  instances).  Given  a  network,  we 
consider  all  static  routes  across  the  routers  to  form  a  single 
routing  instance  and  assume  that  this  instance  is  also  free  of 
instabilities.  Finally,  we  assume  that  routing  instances  are 
not  deployed  in  an  overlay  fashion.  A  routing  protocol  in¬ 
stance  k  is  deployed  in  overlay  between  two  routers  A,  B 
when  A.k  learns  a  route  pointing  to  B  as  the  next-hop  but 
B  is  not  directly  connected  to  A,  and  A  needs  to  rely  on  a 
routing  instance  instance  different  than  k  to  reach  B.  While 
overlay  networks  (e.g.,  an  iBGP  mesh)  can  result  in  routing 
anomalies  [20],  [9],  they  are  beyond  the  scope  of  this  paper. 

Griffin  et  al.  [18]  showed  that  the  absence  of  dispute 
wheels  guarantees  the  convergence  of  the  route  exchanges. 
This  condition  is  a  major  result  and  has  steered  much  of  the 
recent  research  in  BGP  stability.  However,  although  im¬ 
portant,  this  result  may  not  be  practical  especially  for  op¬ 
erators  who  need  to  configure  a  network  with  multiple  IGP 
instances.  The  relationships  between  BGP  networks  (cus¬ 
tomer,  provider,  peer)  form  a  hierarchy  between  them  and 
guarantee  the  absence  of  dispute  wheels  [17].  However, 
routing  protocol  instances  within  a  network  do  not  present 
similar  relationships  nor  patterns  [25].  There  is  currently  no 
guideline  on  how  to  assign  the  AD  values  to  guarantee  the 
safety  of  route  selections.  As  such,  we  propose  the  following 
guideline. 


Guideline  1:  For  a  destination  prefix  P,  all  processes  of 
a  routing  instance  shall  share  the  same  AD  value  and  ev¬ 
ery  routing  instance  shall  be  assigned  a  globally  unique  AD 
value. 

Theorem  1:  Guideline  1  guarantees  the  absence  of  dis¬ 
pute  wheels  spanning  multiple  routing  instances  and  thus  the 
convergence  of  the  route  selections. 

Proof:  We  prove  it  by  contradiction.  Assume  that  a  net¬ 
work  compliant  with  Guideline  1  still  contains  a  dispute  wheel 
n  =  ( U ,  Q,  R)  of  size  k  and  spanning  at  least  two  distinct 
routing  instances. 

Step  1  By  definition  of  the  dispute  wheel,  each  router  it,; 
receives  at  least  two  paths  ( Qi ,  and  -RjQi+i)  to  the  origin. 
Each  of  these  paths  may  have  been  received  through  multi¬ 
ple  routing  processes.  Considering  all  routing  processes  at 
Ui  that  offer  the  path  RiQi+i  (respectively,  Qf),  let  ut.pi 
(respectively,  Ui.p'f)  represent  the  routing  process  with  the 
lowest  AD  value.  By  definition  of  the  dispute  wheel,  Qi  is 
less  preferred  than  RiQi+ 1  at  Ui.  Let  AD(u,i.pi )  be  the  AD 
value  of  the  routing  process  Ui.pi.  We  derive  that  for  all 
*  G  [0;  k  -  1] 

AD  (u.i.p'f)  >  AD{ui.pi)  (1) 

Step  2  Because  there  is  no  configured  route  redistribution, 
and  no  routing  instance  is  deployed  in  overlay,  for  two  suc¬ 
cessive  routers  x  and  y  on  a  forwarding  path  to  the  origin, 
the  set  of  routing  instances  through  which  x  learns  the  route 
is  always  a  subset  of  those  for  y.  As  such,  the  fact  that  it,; 
learns  RiQi+ 1  from  pi  implies  that  the  router  ui+i  (on  the 
path  RiQi+ 1)  is  also  running  a  routing  process  in  (>, ,  and 
|_i  learned  the  subpath  Qi+ 1  from  at  least  pt.  By  definition 
of  the  dispute  wheel,  Qi+i  is  less  preferred  than  II,  +i  Q  i+ 2 
at  |_i.  Therefore,  we  derive  that  for  all  i  G  [0;  k  —  1] 

AD(ui+1.pi)  >  AD[ui+1.pi+i)  (2) 

Step  3  Since  the  network  complies  with  Guideline  1,  all 
routing  processes  within  the  same  routing  instance  have  the 
same  AD  value.  Therefore,  for  every  i  G  [0;  k  —  1]  (modulo 

k) 

AD(ui.pf)  =  AD(  Ui+l-Pi  )  (3) 

From  equations  (2)  and  (3),  we  derive 

AD(ui.p0)  >  AD(ux.pi)  = 

AD(ll2-Pl)  >  AD(U2-P2)  = 

...  >  . . .  = 

AD(w0.pfc_i)  >  AD(u0.p0)  =  AD(Mi.p0) 

Since  the  network  complies  with  Guideline  1,  every  routing 
instance  is  assigned  a  globally  unique  AD  value.  As  such, 
from  the  previous  equations,  we  derive 

Po  =  Pi  =  ■■■  =  Pk-i  (4) 
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Step  4  From  Steps  1  and  2,  for  all  i,  u,  learns  the  path  Qi 
from  two  routing  instances:  pp  and  p, _ -| .  By  definition  of 
Pi t ,  we  derive  that 

AD{ui.p'i )  <  AD(ui.pi- 1) 

Then,  from  equation  (4),  since  p,_-|  =  pi,  we  obtain 

AD{ui.p'i )  <  AD{m.pi) 

Finally,  combining  with  equation  (1),  we  conclude  that 

AD{ui.pi)  <  AD{ui.p'i)  <  AD(ui.pi) 

which  means  that  for  all  i  £  [0;  k  —  1],  pi  =  p\.  Therefore, 
Po  =  Pi  =  ■■■  =  Pk-i  =  Po  =  Pi  =  ■  ■  ■  =  Pk-i-  This  con¬ 
tradicts  the  initial  assumption  that  the  dispute  wheel  spans 
multiple  distinct  routing  instances.  □ 

In  addition  to  guaranteeing  convergence.  Guideline  1  also 
guarantees  loop-free  forwarding  paths. 

Theorem  2:  Guideline  1  guarantees  that  the  forwarding 
paths  between  the  routing  instances  are  devoid  of  permanent 
forwarding  loops. 

Proof:  Again  by  contradiction.  Suppose  a  network 
compliant  with  Guideline  1  contains  a  permanent  loop 
uqUi  . .  .Uk-iUo  for  a  destination  prefix  P.  Let  pt  denote 
the  routing  instance  from  which  Uj  learns  its  active  route  to 
P.  Packet  forwarding  within  each  routing  instance  is  free  of 
instabilities.  As  such,  there  exists  i  £  [0;  k  —  1]  such  that  Ui 
and  )_i  learn  their  active  route  to  P  from  two  distinct  rout¬ 
ing  instances.  Without  loss  of  generality,  we  can  assume  that 
Mo  and  Mi  learn  their  active  route  from  two  different  routing 
instances  po  and  pi  ■  Note  that  the  discussion  below  is  with 
respect  to  destination  P. 

Step  1  This  section  assumes  no  configured  route  redistri¬ 
bution.  Therefore,  mo  learns  its  active  route  from  another 
member  of  routing  instance  po-  This  section  also  assumes 
that  routing  instances  are  not  deployed  in  an  overlay  fash¬ 
ion.  As  such,  the  router  to  which  mo  forwards  its  traffic  (i.e., 
Mi)  is  the  router  that  advertised  the  route  to  uq  through  rout¬ 
ing  instance  po-  We  conclude  that  u.  \  is  also  a  member  of  po- 
However,  the  active  route  at  mi  is  learned  from  a  different 
routing  instance  p1.  Consequently, 

AD(ui,p0)  >  AD(ui,pi)  (5) 

Step  2  For  every  i  £  [1;  k  —  1],  the  router  Ui  points  to  m,;_|_i  as 
its  next-hop.  As  above,  m,+i  is  a  member  of  p*.  Now,  Mi+i’s 
active  route  can  be  learned  from  the  same  routing  instance 
(i.e.,  p.j_|_i  =  pA  or  from  a  different  routing  instance  (pi+i  ^ 
Pi).  We  will  show  that  in  both  cases,  we  have 

AD(uz+i,pi)  >  AD(ui+i,pi+i)  (6) 

Case  1:  p,  =  p,+1 .  Since  the  network  complies  to  Guideline 
1,  all  routing  processes  of  a  routing  instance  have  the 


same  AD  value.  We  conclude  that 

AD(ui+i,pi)  =  AD(ui+i,  pi+i) 

Case  2:  pi  pi+i-  Since  m*+i  learns  its  active  route  from 
Pi+i,  we  derive 

AD(ui+i,pi)  >  AD(ui+i,  pi+i) 

Step  3  Because  the  network  complies  with  Guideline  1,  all 
routing  processes  within  the  same  routing  instance  have  the 
same  AD  value.  In  other  words,  for  every  i  £  [1;  k]  (modulo 

k) 

AD(ui+i,pl+i)  =  AD(ui+2,  Pi+i)  (7) 
From  equations  (5),  (6)  and  (7), 

AD{ui,pf)  >  AD{ui1  pi)  = 

AD(u2,pi)  >  AD{u21pf)  = 

...  >  . . .  = 

AD(u0lPk- i)  >  AD(u0,p0)  =  AD(u!,p0) 

In  particular,  AD{ui,pf)  =  AD(u\,  pf).  As  p0  and  pi  are 
distinct,  this  equation  contradicts  Guideline  1  which  states 
that  every  routing  instance  is  assigned  a  globally  unique  AD 
value.  □ 

Surprisingly,  the  proposed  guideline  has  not  been  reported 
by  the  operational  community  despite  its  conceptual  simplic¬ 
ity.  It  shows  the  value  of  the  type  of  analysis  carried  out  in 
this  paper.  It  is  unclear  this  guideline  can  accomodate  all 
existing  operational  requirements.  We  leave  this  question  to 
future  work.  Finally,  in  prior  work  [24],  we  derived  similar 
guidelines  for  route  redistribution. 

4.  INTERPLAY  BETWEEN  ROUTE 
SELECTION  AND  REDISTRIBUTION 

The  previous  section  disclosed  routing  anomalies  caused 
by  route  selection  alone,  and  identified  a  configuration  guide¬ 
line  for  safe  route  selection.  This  section  analyzes  new  in¬ 
stabilities  that  can  occur  when  route  redistribution  is  used 
in  conjunction  with  route  selection.  The  focus  is  to  explain 
why  the  interplay  between  route  selection  and  route  redistri¬ 
bution  can  easily  cause  the  nondeterministic  forwarding  path 
problem  described  in  Section  1 . 

Section  4. 1  illustrates  the  anomaly,  its  severe  consequences, 
and  examines  the  extensiveness  of  the  problem.  We  present 
experimental  results  which  show  that  the  problem  is  not  just 
specific  to  one  software  implementation  nor  specific  to  one 
combination  of  routing  protocols. 

Section  4.2  hypothesizes  possible  causes  for  the  observed 
anomaly  based  on  additional  experimental  results.  We  pos¬ 
tulate  that  the  root  of  the  problem  is  the  lack  of  a  precise 
specification  of  route  selection  and  route  redistribution,  and 
more  importantly,  how  the  two  procedures  should  interact. 

Finally,  Section  4.3  investigates  how  to  eliminate  the  non¬ 
deterministic  behaviors.  We  present  a  precise  functional  model 
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Figure  5:  Nondeterministic  forwarding  paths. 


for  guiding  the  implementation  of  route  selection  and  route 
redistribution  procedures.  We  formally  establish  the  correct¬ 
ness  and  utility  of  the  model  by  showing  that  an  implemen¬ 
tation  compliant  to  this  model  will  guarantee  both  of  the  es¬ 
sential  properties  defined  in  Section  2  at  all  times. 

4.1  Nondeterministic  Routing  Behaviors 

4.1.1  Motivating  Scenario 

The  following  scenario  is  first  described  in  [7].  Consider 
the  network  depicted  in  Figure  5.  It  consists  of  a  provider 
network  offering  Internet  service  to  a  customer  network 
through  two  IP  links:  A-X  and  B-X.  The  routers  A,  B,  and 
C  in  the  provider  network  run  an  IGP  and  in  addition,  form  a 
full  iBGP  mesh.  At  routers  A  and  B,  static  routes  pointing  to 
prefixes  inside  the  customer’s  network  are  redistributed  into 
BGP  so  that  they  can  be  further  propagated  to  other  BGP  net¬ 
works.  Suppose  that  the  customer  has  designated  the  A-X 
link  as  the  primary  entry  way  for  traffic  arriving  from  the  ser¬ 
vice  provider,  and  B-X  as  a  backup  link.  As  such,  the  BGP 
process  at  router  B  is  configured  with  a  lower  AD  value  (i.e., 
higher  preference)  than  static  routes.  The  expected  behavior 
is  that  whenever  the  A-X  link  is  up  and  A  is  reachable  from 
B,  B  will  forward  all  traffic  to  the  customer  network  via  A 
using  the  route  offered  from  the  BGP  process. 

However,  it  was  reported  that  the  forwarding  paths  at  B 
surprisingly  depend  on  the  timing  of  when  the  static  routes 
are  entered  at  routers  A  and  B  [7].  Such  a  nondeterministic 
behavior  is  clearly  an  anomaly  with  severe  consequences. 
Contrary  to  the  design  goal,  B  may  forward  traffic  to  the 
customer  directly  to  X  and  announce  this  backup  route  to 
other  BGP  neighbors  even  though  the  primary  link  A-X  is 
up  and  accessible. 

To  verify  the  report  of  [7]  ,  we  have  implemented  the 
topology  of  Figure  5  using  4  Cisco  3600  routers  with  IOS 
Version  12.2.  We  have  observed  the  following  behaviors  at 
router  B  regarding  a  particular  prefix  in  the  customer  network: 

Case  1:  When  an  iBGP  route,  redistributed  from  a  static 
route  that  has  been  entered  at  router  A ,  is  the  only  route 
presented  to  the  route  selection  procedure,  it  becomes 
the  active  route.  Then,  when  a  static  route  for  the  same 
prefix  is  installed  locally  at  B,  the  iBGP  route  remains 
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Figure  6:  Experiment  setup.  A  router  is  configured  with 
two  routing  processes  (u,  v )  and  receives  two  routes  to  the 
same  prefix.  Mutual  route  redistribution  is  configured 
between  u  and  v. 

the  active  route  because  of  its  lower  AD  value.  This  is 
the  expected  and  correct  behavior. 

Case  2:  When  the  local  static  route  is  installed  before  the 
iBGP  route  from  router  A  arrives,  the  static  route  be¬ 
comes  the  active  route  and  is  locally  redistributed  into 
BGP.  Then,  when  the  iBGP  route  from  A  arrives,  even 
though  the  newly  received  iBGP  route  has  a  lower  AD 
value  than  the  local  static  route,  the  static  route  re¬ 
mains  the  active  route.  This  is  an  incorrect  behavior 
because  it  violates  property  PI  as  stipulated  in  Section 
2.1.  The  route  with  a  lower  AD  value  did  not  become 
the  active  route. 

4.1.2  Extensiveness  of  Problem 

We  have  conducted  more  experiments  to  examine  the  ex¬ 
tent  of  the  nondeterministic  behavior  described  above.  We 
seek  to  determine  whether  the  anomaly  is  specific  to  one 
software  implementation  (i.e.,  IOS  12.2)  or  one  routing  pro¬ 
tocol  (i.e.,  iBGP). 

The  experiments  have  a  simple  setup  as  depicted  in  Figure  6, 
where  a  single  border  router  connects  two  routing  instances. 
We  have  experimented  with  four  different  implementations 
for  the  border  router:  Cisco  3600  IOS  version  12.2(24a), 
Quagga  Software  Routing  Suite  [4]  version  0.98.6  (which 
is  the  latest  stable  release),  Quagga  Software  Routing  Suite 
version  0.99.10  (latest  unstable  release),  and  and  XORP  [5] 
version  1.4  (latest  release  at  the  time  of  the  experiments). 

For  each  implementation,  two  routing  processes  u  and 
v  are  configured  on  the  border  router.  One  of  the  routing 
processes  is  configured  with  a  lower  AD  value  to  provide 
the  primary  routes  for  all  destinations.  The  other  routing 
process  should  only  provide  backup  routes.  We  have  ex¬ 
perimented  with  different  protocol  combinations  for  these 
processes.  All  the  combinations  are  given  in  the  first  two 
columns  of  Table  1 .  Mutual  route  redistributions  are  config¬ 
ured  between  u  and  v. 

In  each  experiment,  we  advertise  two  routes  to  a  same 
destination  prefix  P  to  the  border  router.  Static  routes  are 
directly  entered  to  the  router  and  the  other  routes  are  adver- 
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Table  1:  Summary  of  experimental  results.  indi¬ 
cates  a  behavior  conforming  to  property  PI  (Section  2.1) 
and  independent  of  the  arrival  order  of  the  advertise¬ 
ments.  “X”  signifies  that  the  arrival  order  of  the  adver¬ 
tisements  impacts  the  outcome  of  route  selection. 


tised  from  a  neighboring  router  running  a  routing  process 
that  peers  with  either  u  or  v.  We  vary  the  timing  of  the  two 
route  advertisements  and  then  inspect  the  forwarding  table 
of  the  border  router  to  determine  the  outcome  of  the  route 
selection  and  route  redistribution  procedures. 

The  results  of  the  experiments  are  summarized  in  Table  1 . 
The  symbol  indicates  a  behavior  conforming  to  prop¬ 
erty  PI  as  stipulated  in  Section  2. 1 :  the  route  with  the  lowest 
AD  value  is  always  selected  as  the  active  route,  independent 
of  the  arrival  order  of  the  advertisements.  The  symbol  “X” 
signifies  that  the  arrival  order  of  the  advertisements  impacts 
the  outcome  of  route  selection  and  there  are  cases  where  the 
route  with  a  higher  AD  becomes  the  active  route. 

We  make  the  following  two  observations.  First,  all  tested 
implementations  produced  unexpected  outcomes  some  of  the 
time.  The  problem  therefore  appears  to  be  pervasive.  Sec¬ 
ond,  the  outcome  varied  from  implementation  to  implemen¬ 
tation  for  some  protocol  combinations.  This  suggests  that 
part  of  the  problem  may  be  due  to  software  coding  errors. 
For  example,  we  discovered  such  an  error  in  the  Quagga 
version  0.98.6  source  code:  When  a  route  is  locally  redis¬ 
tributed  into  the  RIP  protocol,  all  RIP  messages  received 
from  the  neighbors  are  in  fact  discarded  independently  of 
the  AD  values.  This  provides  a  good  explanation  of  the  ob¬ 
served  outcomes  with  RIP  when  using  the  Quagga  0.98.6 
implementation.  However,  given  the  pervasiveness  of  the 
problem,  it  seems  more  logical  to  conclude  that  the  problem 
is  not  entirely  due  to  implementation  errors  but  comes  from  a 
lack  of  a  precise  model  to  understand,  reason  and  support  the 
interactions  between  route  selection  and  route  redistribution. 

4.2  Analysis  of  Root  Cause 

It  is  difficult  to  pinpoint  the  root  cause  or  causes  of  the  ob¬ 
served  anomalies  because  of  the  inaccessibility  to  the  source 
code  of  the  commercial  implementations  and  the  scarce  doc¬ 
umentation  on  this  topic.  In  the  following,  we  try  to  infer  the 
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Figure  7:  An  incorrect  model  of  the  dependencies  be¬ 
tween  RS  and  RR.  A  particular  sequence  of  events  would 
cause  routes  with  higher  AD  values  to  be  selected. 

root  cause  for  the  Cisco  implementation  indirectly  from  in¬ 
formation  available  to  us  from  the  experiment  described  in 
Section  4.1.1. 

As  suggested  by  [7],  a  look  at  the  protocol  specific  routing 
information  bases  (RIBs)  of  router  B  shows  that  the  redis¬ 
tributed  local  static  route  is  present  in  the  RIB  of  the  BGP 
routing  process.  This  discovery  suggests  that  the  Cisco  im¬ 
plementation  may  have  followed  an  incorrect  model  of  the 
dependencies  between  route  selection  and  route  redistribu¬ 
tion  which  is  depicted  in  Figure  7.  This  model  would  cause 
a  violation  of  route  selection  property  (PI)  for  the  scenario 
of  Figure  5  under  this  sequence  of  events: 

t  \  When  the  local  static  route  is  installed  before  the  iBGP 
route  from  router  A  is  received,  the  static  route  is  selected 
to  be  the  active  route  and  redistributed  into  BGP. 

t,2  When  the  iBGP  route  from  A  arrives  at  B,  it  is  put  into 
the  same  RIB  as  the  redistributed  local  static  route.  This 
is  confirmed  by  an  inspection  of  the  BGP  RIB. 

ts  The  BGP  best  path  selection  algorithm  selects  the  best 
route  among  all  the  ones  present  in  the  BGP’s  RIB.  By 
default,  locally  redistributed  routes  are  assigned  a  WEIGHT 
value  larger  than  that  of  routes  received  from  BGP  peers. 
WEIGHT  is  a  Cisco-specific  attribute  [10]  and  for  Cisco 
routers  it  is  the  first  factor  considered  by  the  BGP  best 
path  selection  algorithm.  A  route  with  a  higher  WEIGHT 
value  is  more  preferred.  Therefore  in  this  case,  the  BGP 
best  path  selection  algorithm  selects  the  locally  redis¬ 
tributed  route  as  its  best  path.  Then,  for  stability  reasons, 
the  BGP  process  will  filter  out  the  route  since  it  was  re¬ 
distributed  from  another  process  to  prevent  it  from  being 
considered  by  the  route  selection  procedure  [27].  Finally, 
the  process  converges  and  the  iBGP  route  from  A ,  despite 
having  a  lower  AD  value  than  the  local  static  route,  does 
not  become  the  active  route  at  router  B. 

To  confirm  this  conjecture,  we  reversed  the  WEIGHT  val¬ 
ues:  locally  redistributed  routes  are  now  assigned  a  lower 
WEIGHT  value  than  the  iBGP  routes.  We  repeated  the  ex¬ 
periment,  and  this  time  the  BGP  selected  the  iBGP  route 
as  expected  and  the  anomaly  went  away.  However,  this  fix 
only  applies  to  scenarios  involving  Cisco  implementations 
of  BGP.  For  example,  it  will  not  eliminate  nondeterministic 
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Figure  8:  Existing  implementations  may  cause  a  viola¬ 
tion  of  Property  P2  as  stipulated  in  Section  2.2.  Here, 
router  C  advertises  a  route  that  is  not  the  active  route 
for  the  prefix. 

routing  behaviors  incurred  by  the  interactions  between  RIP 
and  OSPF. 

The  model  depicted  in  Figure  7  can  also  fully  explain  the 
observed  behavior  when  the  iBGP  route  is  received  before 
the  local  static  route  is  installed,  i.e..  Case  1  of  Section  4.1.1. 
The  iBGP  route  becomes  the  active  route  upon  arrival.  Later, 
when  the  static  route  is  installed,  both  routes  are  considered 
by  the  route  selection  procedure.  The  iBGP  route  remains 
the  active  route  because  it  presents  a  lower  AD  value.  The 
static  route  is  not  redistributed  to  BGP  because  it  is  not  the 
active  route  and  the  process  converges  as  expected. 

To  summarize,  we  postulate  that  the  nondeterministic  rout¬ 
ing  behaviors  are  prevalent  because  of  an  incorrect  func¬ 
tional  model  where  locally  redistributed  routes  are  recon¬ 
sidered  as  inputs  to  the  routing  protocol  specific  path  selec¬ 
tion  algorithms  (e.g.,  BGP  best  path  selection  algorithm). 
The  negative  impact  of  this  kind  of  error  is  much  bigger 
than  a  typical  software  bug.  Next,  we  substantiate  this  point 
by  showing  that  an  implementation  based  on  the  incorrect 
model  can  cause  additional  unexpected  outcomes  by  violat¬ 
ing  the  Route  Redistribution  Property  (P2). 

Additional  anomaly:  Consider  the  network  shown  in 
Figure  8. 

t\  Router  C  receives  two  routes  to  the  same  prefix  P  from  a 
BGP  neighbor  (A)  and  a  RIP  peer  ( B ).  Suppose  that  the 
route  from  RIP  becomes  the  active  route  because  the  RIP 
process  has  been  configured  with  a  lower  AD  value. 

t-2  We  assume  that  redistribution  from  RIP  into  BGP  is  con¬ 
figured  at  C.  As  such,  the  active  route  from  RIP  is  re¬ 
distributed  into  BGP.  The  BGP  RIB  contains  two  routes 
to  P:  the  locally  redistributed  route,  and  the  BGP  route 
from  A.  Suppose  because  of  local  policies  (e.g.,  route- 
map  setting  a  larger  WEIGHT  value  to  routes  received 
from  BGP  neighbors)  the  BGP  best  path  selection  algo¬ 
rithm  prefers  the  route  from  the  BGP  neighbor  to  the  lo¬ 
cally  redistributed  route. 

ts  As  such,  router  C  advertises  the  BGP  route  received  from 
its  BGP  neighbor  (A)  to  other  BGP  neighbors  (e.g.,  D) 
instead  of  the  active  route,  i.e.,  the  locally  redistributed 


route.  This  violates  Property  P2  and  may  cause  deflec¬ 
tions  and  permanent  forwarding  loops  as  illustrated  in 
Section  3.2. 

4.3  A  New  Functional  Model  Making  Depen¬ 
dencies  Unambiguous 

This  section  presents  a  solution  framework  to  eliminate 
the  nondeterministic  behaviors.  The  key  element  is  a  func¬ 
tional  model  of  route  selection  and  route  redistribution  that 
makes  the  dependencies  between  the  two  procedures  unam¬ 
biguous  and  guarantees  both  the  route  selection  and  route 
redistribution  properties  as  defined  in  Section  2. 

Section  4.3.1  describes  a  potential  solution  for  vector  pro¬ 
tocols.  Then,  Section  4.3.2  extends  the  proposed  solution 
to  accommodate  link-state  protocols.  The  need  for  exten¬ 
sion  comes  from  the  differences  in  these  two  types  of  routing 
protocols.  While  vector  protocols  first  process  the  received 
information  and  only  advertise  the  best  paths,  link-state  rout¬ 
ing  protocols  relay  all  the  received  information,  even  before 
computing  the  best  paths.  These  characteristics  require  dif¬ 
ferent  designs.  Finally,  Section  4.3.3  shows  that  the  pro¬ 
posed  functional  model  guarantees  the  two  properties  given 
in  Section  2. 

4.3.1  A  Functional  Model  for  Vector  Protocols 

The  proposed  solution  for  vector  protocols  is  depicted  in 
Figure  9  (upper  part).  Each  vector  routing  process  (e.g.,  RIP, 
EIGRP)  is  assigned  two  RIBs:  RIBin  for  incoming  route  an¬ 
nouncements  and  RIBout  for  outgoing  advertisements.  A 
new  announcement  from  a  peer  must  first  pass  through  some 
filters.  The  filters  discard  invalid  advertisements  and  routes 
not  compliant  with  local  policies.  For  example,  RIP  routes 
whose  metric  exceeds  16  are  filtered.  After  passing  the  fil¬ 
ters,  all  routes  are  stored  in  the  RIBin.  A  protocol  specific 
route  determination  algorithm  subsequently  chooses  the  most 
preferred  route  among  all  routes  to  the  same  prefix. 

Then,  each  routing  process  presents  its  best  route  to  the 
route  selection  procedure.  The  active  route  is  selected  based 
on  the  AD  values  and  installed  in  the  router’s  FIB. 

The  router’s  FIB  maintains  the  routes  that  are  used  to  for¬ 
ward  traffic.  In  this  model,  an  active  route  is  by  default  re¬ 
distributed  into  the  RIBout  of  the  selected  process.  For  ex¬ 
ample,  if  the  active  route  comes  from  routing  process  A.k, 
then  the  active  route  is  by  default  installed  into  the  RIBout  of 
routing  process  A.k  and  advertised  to  the  peer  processes  of 
A.k  in  routing  instance  k.  The  active  route  may  also  be  redis¬ 
tributed  into  other  routing  processes  according  to  the  route 
redistribution  configuration  on  the  router.  Routing  policies 
can  be  applied  every  time  an  active  route  is  redistributed. 

In  this  model,  a  locally  redistributed  route  is  not  consid¬ 
ered  by  any  of  the  protocol  specific  route  determination  al¬ 
gorithms.  As  such,  the  status  of  this  route  is  unambiguous 
from  the  perspective  of  the  route  selection  procedure. 
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Figure  9:  A  functional  model  of  RS  and  RR  supporting  all  protocols. 


4. 3. 2  Extension  for  Link  State  Protocols 

This  section  extends  the  vector  model  to  accommodate 
link-state  protocols.  As  depicted  in  Figure  9  (lower  part), 
each  link-state  routing  process  is  also  associated  with  two 
databases:  a  RIB  and  an  Eligible  Information  Base  (EIB). 
The  RIB  stores  the  regular  link-state  updates,  including  lo¬ 
cally  redistributed  routes.  All  members  of  one  link-state 
routing  instance  will  eventually  have  identical  information 
in  their  RIBs.  Content-wise,  EIB  is  a  subset  of  RIB.  It  is 
a  separate  entity,  used  to  isolate  the  routes  that  are  eligi¬ 
ble  to  become  active  at  the  router.  An  additional  built-in 
filter  between  RIB  and  EIB  prevents  locally  redistributed 
routes  from  entering  the  EIB.  Then,  the  protocol  specific 
route  determination  algorithm  is  executed  based  on  the  EIB 
and  the  best  route  is  presented  to  the  route  selection  proce¬ 
dure.  Again,  there  is  no  ambiguity  from  the  perspective  of 
the  route  selection  procedure. 

The  route  redistribution  part  accordingly  requires  a  simple 
extension.  When  the  target  routing  process  is  a  link-state 
protocol,  the  redistributed  route  is  inserted  into  the  RIB  of 
the  target  routing  process. 

4.3.3  Correctness  of  the  Proposed  Model 

The  following  theorem  establishes  the  correctness  of  our 
model. 

Theorem  3:  The  proposed  model  guarantees  route  selec¬ 
tion  property  PI  and  route  redistribution  property  P2  that 
are  presented  in  Section  2. 

Proof:  Consider  a  router  and  a  destination  prefix  P.  We 
first  prove  by  induction  that  independent  of  the  message  ar¬ 
rival  order,  property  PI  is  guaranteed,  i.e.,  the  route  with  the 
lowest  AD  value  is  selected  as  the  active  route. 

1.  PI  is  trivially  satisfied  when  no  route  to  P  is  offered. 

2.  We  assume  that  at  time  ti,  the  RS  and  RR  procedures  have 
converged  and  PI  holds  true.  Now,  suppose  the  first  routing 
event  after  1 1  occurs  at  time  t  p.  a  route  is  withdrawn  or  a  new 
route  (either  static  or  coming  from  a  peer)  is  added.  In  the 
case  of  a  route  withdrawal,  the  RIBin(s)  and  the  EIB(s)  of  the 


remaining  routing  processes  that  still  have  a  route  are  not  im¬ 
pacted  by  the  route  withdrawal.  Each  routing  process  selects 
its  most  preferred  route  and  presents  it  to  the  route  selection 
algorithm.  The  latter  chooses  the  route  with  the  lowest  AD 
value.  The  active  route  may  be  redistributed  into  different 
routing  processes,  but  this  does  not  affect  the  RIBin(s)  and 
the  EIB(s).  As  such,  the  process  converges  and  PI  remains 
true.  In  the  case  of  a  new  route,  contrary  to  existing  imple¬ 
mentations,  locally  redistributed  routes  are  excluded  from 
consideration  by  any  of  the  protocol  specific  route  determi¬ 
nation  algorithms.  This  eliminates  the  error  condition  de¬ 
scribed  in  Section  4.2.  Consequently,  each  routing  process 
that  possesses  a  non  empty  set  of  routes  to  P  presents  its 
most  preferred  one  to  the  route  selection  procedure,  and  the 
route  with  the  lowest  AD  value  is  then  selected  to  become 
the  active  route.  Again,  the  active  route  may  be  redistributed 
into  different  routing  processes,  but  this  does  not  modify  the 
content  of  the  RIBin(s)  and  the  EIB(s).  As  such,  the  process 
converges  and  PI  remains  true. 

We  have  shown  that  the  model  guarantees  PI.  Further¬ 
more,  in  the  proposed  model,  a  vector  routing  process  ad¬ 
vertises  routes,  to  its  peers,  from  the  RIBout.  As  such,  a 
vector  routing  process  advertises  a  route  only  if  active.  In 
addition,  routes  are  redistributed  directly  from  the  router’s 
FIB.  Therefore,  a  route  can  be  redistributed  only  if  active. 
The  model  guarantees  P2.  □ 

5.  RELATED  WORK 

Router  vendors  [13]  mention  that  route  selection  can  cause 
forwarding  loops  but  do  not  provide  any  illustration  nor  guide¬ 
line  to  avoid  them.  Some  documents  [12],  [11],  and  [24]  ex¬ 
posed  instabilities  due  to  route  redistribution.  In  earlier  work 
[27],  we  developed  a  framework  to  reason  about  the  impacts 
of  route  redistribution  at  a  network-wide  level.  Yet,  this  pa¬ 
per  shows  that  route  selection  by  itself,  and  its  interplay  with 
route  redistribution,  can  also  be  the  source  of  routing  anoma¬ 
lies.  Our  work  is  the  first  to  illustrate  how  routing  instabil¬ 
ities  may  result  from  route  selection  alone  and  its  interplay 
with  route  redistribution.  We  also  analyze  the  root  causes  of 
these  instabilities  and  develop  guidelines  and  solutions  for 
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preventing  them. 

Several  studies  [32],  [20]  looked  at  the  interactions  be¬ 
tween  BGP  and  its  underlying  IGP  and  revealed  potential 
instabilities.  In  comparison  to  these  studies,  the  scope  of  our 
work  is  much  broader.  Although  our  study  does  not  encom¬ 
pass  overlay  routing  protocols,  we  show  that  the  interactions 
between  any  two  routing  processes,  regardless  which  proto¬ 
cols  they  run,  can  create  routing  anomalies  and  the  instabil¬ 
ities  are  not  limited  to  route  oscillations  and  loops. 

6.  CONCLUSION  AND  FUTURE  WORK 

We  have  demonstrated  that  the  interactions  between  rout¬ 
ing  protocols  are  a  much  more  complex  problem  than  pre¬ 
viously  believed.  While  it  has  been  recognized  that  route 
redistribution  (RR)  can  easily  cause  routing  anomalies  and 
should  be  handled  with  care,  this  paper  shows  that  route  se¬ 
lection  (RS)  in  itself,  and  its  interplay  with  RR,  can  also  re¬ 
sult  in  a  wide  range  of  routing  anomalies.  It  establishes  a 
strong  link  between  RS  and  RR  and  some  of  the  puzzling 
routing  anomalies  discovered  in  operational  networks. 

The  overall  results  suggest  a  twofold  conclusion.  On  the 
one  hand,  the  news  is  somewhat  bleak.  The  RS  and  RR  pro¬ 
cedures  are  highly  susceptible  to  routing  anomalies  and  the 
range  of  anomalies  is  much  wider  than  previously  reported. 
Our  study  revealed  that  all  tested  implementations  have  in¬ 
correctly  represented  the  dependencies  between  RS  and  RR. 
The  lack  of  a  well  defined  standard  for  these  procedures 
has  certainly  compounded  the  problem.  On  the  other  hand, 
this  paper  shows  that  it  might  be  possible  to  mitigate  the 
instabilities  through  a  deeper  understanding  of  the  problem. 
Many  well-formulated  theoretical  frameworks  have  been  de¬ 
veloped  for  existing  protocols,  particularly  for  BGR  Because 
of  its  severity  and  prevalence,  this  problem  deserves  similar 
attention  from  the  networking  community. 

In  the  big  picture,  we  also  see  the  need  for  ongoing  ef¬ 
forts  aimed  at  redesigning  the  Internet  routing  architecture 
[1],  [23],  [3]  to  closely  examine  the  role  of  the  interactions 
between  routing  instances.  The  correctness  of  individual 
routing  protocols  may  still  not  be  sufficient  to  guarantee  cor¬ 
rect  routing  in  those  settings.  The  current  RS  and  RR  pro¬ 
cedures  were  invented  without  much  consideration  given  to 
their  safety  properties.  A  clean  slate  redesign  of  these  pro¬ 
cedures,  with  an  emphasis  on  robustness,  should  be  highly 
desirable.  [21]  takes  a  first  step  in  this  direction.  It  intro¬ 
duces  an  elegant  framework  allowing  operators  to  define  and 
reason  about  routing  protocols  and  their  interactions.  In  the 
future,  we  hope  to  build  upon  such  frameworks  to  design  the 
next  generation  of  “glue  logic”  [25]  for  routing  protocols. 
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